Syntagmatic, Paradigmatic, and Automatic N-Gram Approaches to Assessing Essay Quality

Computational indices related to n-gram production were developed in order to assess the potential for n-gram indices to predict human scores of essay quality. A regression analyses was conducted on a corpus of 313 argumentative essays. The analyses demonstrated that a variety of n-gram indices were highly correlated to essay quality, but were also highly correlated to the number of words in the text (although many of the n-gram indices were stronger predictors of writing quality than the number of words in a text). A second regression analysis was conducted on a corpus of 88 argumentative essays that were controlled for text length differences. This analysis demonstrated that ngram indices were still strong predictors of essay quality when text length was not a factor. Writing Practice and Assessment Writing is a critical skill related to academic and professional success (Kellogg & Raulerson, 2007). However, large-scale assessments often show that writing proficiently is difficult for many students (National Commission on Writing, NCW, 2003). One method from which to evaluate effective writing and better understand writing quality is to examine human judgments of writing quality (i.e., the scores assigned to a writing sample by trained readers). These scores can have important consequences to the writer. Such consequences are especially evident in the values attributed to writing samples used in student evaluations (i.e., class assignments) and high stakes testing such as the Scholastic Aptitude Test and the Graduate Record Examination. Once a better understanding of writing quality is reached, opportunities for extended practice guided by individual feedback can be given to students in targeted areas. However, teachers are often limited in their Copyright © 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. opportunities to provide feedback on student writing due to limited time and large class sizes (National Commission on Writing, 2003). One solution has been the use of automated essay scoring (AES) systems. AES systems utilize sophisticated software to evaluate the structure, content, and overall quality of written samples (Shermis & Burstein, 2003). By automating portions of the grading and feedback process, students have more opportunities for writing practice, with fewer burdens placed on instructors. Automated Essay Scoring As noted above, AES systems allow students to practice writing and receive feedback, without adding to teachers’ burdens (Dikli, 2006). Writing can be assessed via combinations of statistical modeling, natural language processing (NLP) tools, artificial intelligence (AI), machine learning techniques, and other similar methods. Systems such as e-rater (Burstein, Chodorow, & Leacock, 2004), IntelliMetric (Rudner, Garcia, & Welch, 2006), and Writing-Pal (W-Pal, Dai, Raine, Roscoe, Cai, & McNamara, 2011) rely primarily on NLP and AI. In these systems, expert readers rate a corpus of essays to identify the overall quality of individual essays. Essays are then automatically analyzed along many linguistic dimensions, and statistically analyzed to extract features that discriminate between higher and lower-quality essays. Finally, weighted statistical models combine the features into algorithms that assign grades to essays. For instance, a study by McNamara, Crossley, and McCarthy (2010) indicated that human judgments of essay quality are best predicted at the linguistic level by linguistic indices related to lexical sophistication (i.e., word frequency and lexical diversity) and syntactic complexity (i.e., the number of words before the main verb). These indices accurately classified 67% of essays as being either of low or high quality. Crossley, McNamara, Weston, and McLain-Sullivan (2011) examined differences Approaches to Assessing Essay Quality 214 Proceedings of the Twenty-Fifth International Florida Artificial Intelligence Research Society Conference